Method for tracking and assessing program participation

ABSTRACT

Disclosed is a computerized decision support system and method for a) tracking participation within programs, b) capturing participant&#39;s participation activity and assessment information in a format that can be easily analyzed and c) distilling the participation and assessment data into useful management and evaluation information. The repository of longitudinal data can be analyzed and reported for case-management and program-evaluation purposes. An assessment module enables analyzable assessment instruments to be custom-defined by the system user, e.g. a program manager. The customized assessment instrument is used to provide answer-restricted questions during an assessment interview, enabling virtually any data item to be tracked historically. The system captures date/time-stamped participation information at various levels of detail and stores this information in a way that can be easily retrieved and analyzed for program and participant-focused analysis. A set of industry-standard participation events can be tracked, with supporting detail, as well as less-data-intensive ad hoc user-defined activities. The data model underlying the system, and the implementation of this model within a relational database system, provides a great degree of flexibility, adaptability and efficient navigation through the database for analysis and reporting. Though numerous program-evaluation reports are provided, a set of intermediary aggregations of data is also available for efficient evaluation of additional program outcome measures.

RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional PatentApplication Serial No. 60/323,008 entitled “PROGRAM PARTICIPATIONTRACKING, ASSESSMENT, AND EVALUATION SYSTEM” filed Sep. 18, 2001,incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] The present invention relates generally to the field of databasemanagement and more specifically to tracking and assessing humansubjects as they progress through a variety of programs.

BACKGROUND OF THE INVENTION

[0003] The need for the invention was first raised in a Juvenile Justicesetting. Juvenile courts, unlike their adult counterparts, focus onrehabilitation of the offender (vs. punishment). As a result, theseagencies need to provide programs, services, and interventions thataddress the assessment and rehabilitation of these young offenders. Someprograms are offered directly by the court (e.g. probation services),but most are provided by outside agencies.

[0004] The goal is for offenders to be assessed and then referred toprograms that hold promise of impacting the youth in a positive way. Itis important that the referrals direct the youth to appropriate programsfor his/her needs (and risk). Assessments are conducted to determine thecharacteristics of the subject (e.g. demographic information such assex, race, age, socioeconomic situation, but also behavioral, physical,psychiatric needs, risks, strengths, weakness, etc.). The candidateprograms, on the other hand, have characteristics (i.e. mission, goals,expertise, target audience, capability, capacity, cost, eligibilityrequirements, etc.) In an ideal world, some exceptionally skilled andinformed case worker or court official would match the youth, havingdocumented characteristics, to the best program(s), having documentedcharacteristics. Making this “matching” decision would also take intoaccount the information “what intervention works best for what kind ofyouth? And are these intervention services offered by the availableprograms?” These are among the criteria used to measure how “good” acandidate match might promise to be.

[0005] Answering these questions is a very complex and data intensivetask, especially with thousands of youth and scores of differentprograms from which to choose. Methods of assessing and characterizingthe youth and programs, documenting the level of participation and theinterventions used, and capturing the outcomes of historical matches areneeded. And, importantly, information technology in the form ofdata-collection, database and analysis tools are needed to enable themethodology.

[0006] Static information about the youth as well as longitudinal anddynamic information about the youth's needs, behaviors, attitudes, etc.,together with longitudinal intervention information related to his/herparticipation in multiple programs, program service-delivery informationand goals, must be captured. Furthermore, this information needs to becaptured in a format that can accommodate very different kinds of datacoming from many different sources. The data-collection method and toolneed to be flexible yet robust.

[0007] The problem of maintaining youth assessment information alone isa daunting task. Assessment instruments (e.g. questionnaires, surveyforms, etc.) vary from program to program. And often there are multipleassessment instruments used within the same program. Frequently,questions are shared by multiple instruments. Similar questions areexpressed inconsistently across instruments (e.g. one expression of thequestion might be in a multiple-choice format, while another might befree format.) The assessment instrument itself is often dynamic, havingquestions added, changed, or deleted over time. The sheer number ofinstruments and information elements is overwhelming, and maintainingsuch instruments within an information system could require major andongoing programming effort to “program them into and then maintain them”within the application.

[0008] There is a growing universal and pressing need for methods andtools to assist in program outcome measurement, and, more generally, toprogram evaluation. This impetus has arisen partially due to thepresence of more and more human service programs and the rise innon-profit initiatives. Also, funders of such programs are demandingaccountability and are expecting to see how their contributions arebeing used. United Way has recently mandated that its member agenciesimplement Program Outcomes Measurement programs and methods, and isactively training these agencies in this practice.

SUMMARY OF THE INVENTION

[0009] Determining how youth are impacted as a result of programinterventions is an important question. While the present invention willbe described in the juvenile justice environment, the present inventionneed not be confined to youth subjects or to a juvenile justice setting.The data model underlying the invention was developed to track anyparticipants in any programs within a robust database structure thatcould support numerous multi-dimensional and parametric program outcomemeasurement objectives. The ease in navigating through the database tomeasure various program outcome indicators is demonstrated in theaccompanying invention description.

[0010] According to a preferred embodiment of the present invention, anaggregate assessment of a group of subjects is performed givensubject-specific assessments. The assessments are conducted usingassessment instruments and data-validation rules. A means ofrepresenting a set of questions, answer restrictions, andquestion-answer-validation rules is provided within anassessment-instrument data structure. A means of representing aplurality of assessment instruments is provided within theassessment-instrument data structure and for each assessment instrumentan assessment-instrument key and a plurality of links to associatedquestions is stored. A means of representing a plurality of subjects isprovided and for each subject a subject key and a plurality of subjectattributes are stored within a subject data structure. A means ofrepresenting a plurality of assessment events is provided within anassessment-event data structure, and for each assessment event, anassociated assessment-event key, subject key and assessment-instrumentkeys, and a reference to a point in time are stored in theassessment-event data structure. Within the assessment-event datastructure, the assessment results including of a plurality of validatedanswers to a plurality of associated said questions for each assessmentevent are stored for a subject group by aggregating the assessmentresults from subjects within the group and utilizing linked data withinthe data structures.

[0011] According to an embodiment of the present invention, programparticipation is tracked within programs that provide services toprogram participants. The participation experience includes serviceevents and said assessment events and the assessment data has beenstored in assessment-instrument and assessment-event data structures. Ameans of representing a plurality of programs having varying programcomponents and services is provided and a unique program key and aplurality of program attributes are stored in a program-definition datastructure. A means of representing a plurality of program-participationexperiences is provided and for each said program-participationexperience a unique program-participation key, an associated programkey, an associated program-participant key, and a reference to aparticipation time period is stored within a program-participation datastructure. A plurality of program-participation activities and eventsfor a given program-participation experience is represented within theprogram-participation data structure including assessment events andeach event is linked to the experience by the program-participation key.

[0012] According to another embodiment of the present invention theeffectiveness of programs that provide services to program participantscan be assessed given assessment questions that have been stored in anassessment-instrument data structure and validated assessment resultsfor program participants that have been stored as assessment events inan assessment-event data structure and analyzed, in aggregate, relativeto program-participation experiences. A means of representing aplurality of programs having varying program components and services isprovided and a unique program key and a plurality of program attributesare stored in a program-definition data structure. A means ofrepresenting a plurality of program-participation experiences isprovided and for each program-participation experience, a uniqueprogram-participation key, an associated program key, an associatedprogram-participant key, and a reference to a participation time periodare stored within a program-participation data structure. For a givenprogram-participation experience, a plurality of program-participationactivities and events, including said assessment events representing,are stored within the program-participation data structure and eachevent is linked to the experience by the program-participation key. Forany program having outcome indicators represented in the assessmentinstruments, the set of said assessment events associated with theinstrument(s) is selected with the program's participants, andprogram-level assessment results are derived by aggregating theassessment results of the assessment events.

[0013] The program participation tracking, assessment, and evaluationsystem of the present invention is an integrated decision support toolthat provides support for case workers, program managers andadministrators in delivery of appropriate and beneficial programservices to program participants.

[0014] The tracking, assessment, and evaluation system of the presentinvention is able to use assessment instruments to capture varied andanalyzable longitudinal information about youth within the juvenilejustice system. According to an embodiment, the system maintainsinformation about programs, services, providers, funding, etc. Thesystem measures the level of participation of the youth within aprogram—the delivery methods and workers involved, the interventionsused, the services received, the contacts that were made, incidencesthat occurred etc. According to an embodiment, the system facilitatesanalysis of the information for case management and program evaluationpurposes.

[0015] The program tracking, assessment, and evaluation system of thepresent invention enables case workers working with program participantsto track participation activity and assessment information about thoseparticipants, and to be able to measure the effectiveness of the programand of program services. It provides a user-friendly interface tocapture critical participation and assessment information.

[0016] According to a feature of the invention, program managers candocument program characteristics and services, monitor the operation ofthe programs, and evaluate program effectiveness. Information can beused to identify problems and opportunities, and support decisionsrelated to poorly used or unnecessary services, problem providers, needfor new or changed services, etc.

[0017] The present invention, embodied as a relational databaseapplication, stores the participation and assessment information in sucha manner as to facilitate the analysis of the captured data for programevaluation as well as case management purposes. The underlying datamodel provides a general and flexible framework for adapting tovirtually any program-participation scenario. The user interface thatsupports the definition, capturing and reporting of assessmentinformation do not involve instrument-specific tables. Rather they relyon instrument-specific rows. Thus, the assessment instrument is definedby the data rather than by the table structures.

[0018] An important byproduct of the flexible data structures is in theease and flexibility of analysis and reporting. Importantly, assessment,participation activity and demographics can be easily combined. Basicreports, designed around user-supplied parameters, can be developed toaccommodate numerous reporting requirements.

[0019] An exemplary embodiment of the present invention includes anassessment module that permits customization of assessment instrumentswithout the aid of a professional software developer. Theuser-customized assessment instruments can then be used to providequestions to be answered in an assessment session or interview. Further,these questions may or may not have associated coded answer choices oranswer restrictions. The answer restrictions enhance the question's orinstrument's ability to be analyzed. Standard industry classificationcodes (e.g. diagnosis, treatment, or criminal codes, etc.) can beimported into the database to provide answer constraints. Other answerchoices may be maintained, through the systems' assessment userinterface, in a central repository of “permissible answers.”

[0020] The present invention provides a flexible method of trackingfundamental program activities. According to a feature of an embodimentof the present invention, each activity is captured with a date, andpossibly time, along with relevant supporting data. Participant-specificreports such as an activity summary and assessment summary can be viewedto provide valuable information about the participant's level ofparticipation and about the impact of the participation in changing theparticipant's attitudes, behaviors, skills, etc. In other words, itprovides information for measuring a participant's progress relative totargeted program outcomes. These fundamental program activities (events)include assessments, worker associations, contacts, services received,etc. In addition, miscellaneous activities can be tracked as well.Additional activity categories can be added by the user, and thentracked.

[0021] Detailed participant-specific activity and assessment data can beaggregated and analyzed at the program, provider, or other aggregatelevel. The longitudinal data can be analyzed to compare before and aftermeasures, or used to evaluate program outcomes vis-a-vis other programs'outcomes.

[0022] In an embodiment, selected assessment-instrument questions can beused as outcome indicators. Analyzing these indicator-type questions istantamount to analyzing the associated indicators. Values for theseindicators, rolled up across multiple assessments, can provide“program-focused” indicator data that can be combined with otherindicator data to assess program effectiveness.

[0023] An aggregate-score assessment instrument, i.e. an instrumentwhose questions are numeric, and can be combined or aggregated into aninstrument-level score, can be designated as an indicator as well.Scores for these aggregate-score instrument indicators can also be usedas input into program outcome measurement.

[0024] In an embodiment, the assessment information can be analyzed incombination with participant demographic and participation activityinformation (e.g. services received, workers associated, contacts made,etc.). The present invention contains many reports that analyzedemographic, participation-activity and assessment information. Eachreport is based upon queries which accept parameters that specify, forexample, the question or instrument to be analyzed, the type of output,the level of detail of the output, the type of evaluation, etc. It canbe contemplated that additional parametric reports can be added to theset currently defined.

[0025] The present invention also creates intermediary tables (partiallyaggregated) that streamline analysis and evaluation of additionalprogram outcome measures.

[0026] The extract files generated by the present invention can beimported into more sophisticated statistical analysis programs (e.g. SASor SPSS) for multivariate or other advanced analysis purposes.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027]FIG. 1 is a block diagram of a network configuration for practiceof an embodiment of the present invention;

[0028]FIG. 2 is a context diagram for an embodiment of the presentinvention;

[0029]FIG. 3a is a data flow diagram for an assessment instrumentdefinition module according to an embodiment of the present invention;

[0030]FIG. 3b is a data flow diagram for a program information moduleaccording to an embodiment of the present invention;

[0031]FIG. 3c is a data flow diagram for a participant informationmodule according to an embodiment of the present invention;

[0032]FIG. 3d is a data flow diagram for a participation recordationmodule according to an embodiment of the present invention;

[0033]FIG. 3e is a data flow diagram for a report preparation moduleaccording to an embodiment of the present invention;

[0034]FIG. 4 is a data flow diagram for a program information moduleaccording to an embodiment of the present invention;

[0035]FIG. 5 is a data model of some basic elements of an embodiment ofthe present invention;

[0036]FIG. 6 is a data model of participant information as structuredaccording to an embodiment of the present invention;

[0037]FIG. 7 is a data model of program offering information asstructured according to an embodiment of the present invention;

[0038]FIG. 8 is a data model of individual participation information asstructured according to an embodiment of the present invention;

[0039]FIG. 9 is a data model of individual assessment information asstructured according to an embodiment of the present invention;

[0040]FIG. 10 is a data model of program evaluation information asstructured according to an embodiment of the present invention;

[0041]FIG. 11 is a data model for program evaluation information asstructured according to an embodiment of the present invention;

[0042]FIG. 12 is a data flow diagram for a program assessment moduleaccording to an embodiment of the present invention;

[0043]FIG. 13 is a data flow diagram for an assessment aggregationmodule according to an embodiment of the present invention;

[0044]FIG. 14 is a data flow diagram for assessment retrieval moduleaccording to an embodiment of the present invention;

[0045]FIG. 15 is a data flow diagram for an answer retrieval moduleaccording to an embodiment of the present invention;

[0046]FIG. 16 is a data flow diagram for score and interpretationretrieval module according to an embodiment of the present invention;

[0047]FIG. 17 is a data flow diagram for an assessment to answer joiningmodule according to an embodiment of the present invention;

[0048]FIG. 18 is a data flow diagram for assessment to demographicprofile joining module according to an embodiment of the presentinvention;

[0049]FIG. 19 is a data model for a navigational path based on programoffering parameters according to an embodiment of the present invention;

[0050]FIG. 20 is a data model for a navigational path based on questionparameters according to an embodiment of the present invention;

[0051]FIG. 21 is a data model for a navigational path based oninstrument parameters according to an embodiment of the presentinvention;

[0052]FIG. 22 is a data model for a navigational path based oninstrument/program offering parameters according to an embodiment of thepresent invention;

[0053]FIG. 23 is a data model for a navigational path based on programoffering/ instrument parameters according to an embodiment of thepresent invention;

[0054]FIG. 24 is a data model for a navigational path based on programoffering/question parameters according to an embodiment of the presentinvention;

[0055] FIGS. 25-46 are examples of user interface screens for populatingdata structures of an embodiment of the present invention;

[0056]FIG. 47 is an illustration of a form representing anaggregate-score instrument according to an embodiment of the presentinvention;

[0057] FIGS. 48-51 are examples of SQL queries for generating programevaluation information according to an embodiment of the presentinvention; and

[0058] FIGS. 52-57 are examples of reports that can be generatingutilizing an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] As illustrated in FIG. 1, a preferred embodiment of theinvention, a Program Participation Tracking, Assessment, and Evaluationsystem, is in the form of a Microsoft Access 2000® database application,running on a Windows NT® local area network. The system is implementedin two files, a User Interface 10 (named PPTAEUserIF.mdb), and aDatabase 12 (named PPTAEData.mdb). Workstations in the network should berunning Windows NT Workstation®\or Windows 98® operating systems. MSAccess 2000® must be installed on each workstation running theapplication, as well as on the NT server. Optionally, a PC-basedstatistical analysis program (e.g. SPSS) can be used to perform advancedanalysis on data exported from the Assessment and Program Evaluationdatabase application. Finally, MS Graph 2000® is required to displaygraphical reports.

[0060] Subsequent figures provide detail about the User Interface 10 andDatabase 12 components of the system. FIGS. 2 through 4 and 12 through18 focus on the User Interface 10 processes, where FIGS. 5 through 11focus on the Database 12 data models. The User Interface 10 is composedof 5 major processes, as illustrated in FIG. 2. Each of the 5 processesis decomposed in FIGS. 3a through 3 e. The complete hierarchicaldecomposition is as follows: Context Diagram   Diagram 1   Diagram 2  Diagram 3   Diagram 4   Diagram 5     Diagram 5.2       Diagram 5.2.1        Diagram 5.2.1.1           Diagram 5.2.1.1.1           Diagram5.2.1.1.2           Diagram 5.2.1.1.3           Diagram 5.2.1.1.4          Diagram 5.2.1.1.5

[0061] User Interface Process Overview

[0062] The processes embodied in the present invention will be describedin five basic modules, outlined in FIG. 2, as follows:

[0063] Assessment-Instrument Maintenance Module

[0064] The first process, Maintain Assessment-Instrument DefinitionInformation 210, is the system module where the assessment instrumentsare maintained. An Assessment Instrument 870 (FIG. 8) is an electronicversion of a questionnaire or form containing questions that are to beanswered by, in this case, a program participant, or by someoneinterviewing the participant and acting on his/her behalf. A case workermight be the most likely respondent. The user interface relative to thisprocess is exemplified in the screens displayed in FIGS. 25 through 31.

[0065] Program Maintenance Module

[0066] The second process, Maintain Program Information 220, is manifestin the system module where information defining the program ismaintained. Various screens collect information about the program. Suchinformation includes the program's missions, goals and eligibilityrequirements to the provider, workers and services provided. This modulepermits the program configuration that must occur before any programparticipant can be tracked.

[0067] Participant Maintenance Module

[0068] A third process, Maintain Participant Information 230, is thesystem module where information about the participant is maintained. SeeFIG. 32 for a sample screen. For the purposes of this description, it isassumed here that the information is primarily static demographicinformation as opposed to the more dynamic participation informationthat is maintained in the fourth process, Record ParticipationInformation, 240. It may happen that much of this information is alreadystored in some other electronic medium, i.e. another computer system. Inthat case, a “refresh” interface can be developed to synchronize theProgram Participation Tracking, Assessment and Evaluation system withparticipant attributes found in both systems. In fact, the preferredembodiment of the invention does include a module to refresh its systemwith demographic, address, criminal history, etc. maintained in a maininformation system.

[0069] Record Participation Module

[0070] The fourth process, Record Participation Information 240,represents the system module where the bulk of the day-to-day entry ofinformation takes place. This user interface provides screens whichcapture a great amount of information relating to a participant'sparticipation in a program. FIGS. 33 through 46 show examples of thisinterface.

[0071] Program Evaluation Module

[0072] A fifth process, Prepare and Output Reports 250, represents thesystem module that compiles information located within the database thatmeets user specified requirements and presents the retrieved informationto the user in a user specified manner. Assess/Evaluate ProgramEffectiveness 410 (FIG. 4), a sub-process of Prepare and Output Reports250, represents the compilation, aggregation, analysis and presentationof participation and related data into useful management and evaluationinformation. Examples of how the database is navigated to deriveprogram-evaluation information is discussed in subsequent figures.

[0073] The Assessment-Instrument Maintenance Module

[0074]FIG. 3a is a decomposition of the Maintain Assessment-InstrumentDefinition Information process 210. The sub-processes represented by311, 312 and 313 describe a user interface that maintains answers,questions, and instruments, respectively.

[0075] Though the focus of this section is on the process MaintainAssessment-Instrument Definition Information 210, it is useful to referto the corresponding section of the data model (FIG. 9) that focuses onIndividual Assessments (as opposed to Program Assessments). Frequentreference will be made to objects in FIG. 9.

[0076] The purpose of the Maintain Answers 311 interface is to provide ameans of maintaining Permissible Answers 950 (FIG. 9) in a generic“answer repository.” The answers in the answer repository willultimately be used to constrain assessment-instrument multiple-choicequestions. FIG. 25 shows the screen in the preferred embodiment wherethe generic answer repository is maintained. Note that the set ofanswers contain industry-neutral answers such as “Yes” and “No”;“Satisfactory” and “Unsatisfactory”; “Increase”, “Decrease” or“NoChange”, as well as industry-specific ones. Since this repository isuser-maintained, it can contain answers that satisfy the assessmentneeds of target programs. For example, “K”, “1^(st)”, “2^(nd)”, etc.might be answers used by programs catering to youth, when, for example,an assessment needs to document the current school grade.

[0077] This repository of manually added answers is one of two types ofanswer domains, the other type being imported industry-standard codes.Diagnostic, treatment, or criminal codes are typical examples. TheDSM-IV diagnostic codes, if imported into the system, could be used toconstrain diagnostic-related questions. Assume, for purposes ofillustration, that the system has imported a table containing healthdiagnostic codes and called it domDSMIV (the “dom” prefix standing for“Domain”). For example, domDSMIV could look like the following: Diag IDDiag Code Diag Description 1 291.8 Alcohol-Induced Anxiety Disorder 2303.90 Alcohol Dependence 3 308.3 Acute Stress Diorder

[0078] Sub-process Maintain Questions 312 is the part of the AssessmentInstrument Maintenance Module where the Questions 930 (FIG. 9) aremaintained. A “question repository” is maintained in much the same waythat an answer repository was maintained by Maintain Answers 311.

[0079] It is presumed that an effort precedes the implementation of thepresent invention that analyzes the numerous in-use instruments andculls from these instruments a core set of non- or minimally overlappingquestions. When multiple similarly worded questions can be re-phrasedinto a normalized standard, the opportunity to perform analysis isimproved. The same question may show up in multiple instruments, and canbe analyzed, if desired, independently of the assessment instrument inwhich it appears. Question-based analysis is covered later when theprocess Assess/Evaluate Program Effectiveness 410 (FIG. 4) is discussed.

[0080] Questions 930 (FIG. 9) are characterized into different typesaccording to the restrictions placed on their answers. These types areshown in FIG. 9 as: MultiChoice 935, Date 945, Freeform 955, Inherited965, and Numeric Range 975.

[0081] Some of these categories are further subdivided. MultiChoicequestions may draw their answers from the generic answer repository orfrom one of the imported code tables (e.g. “domDSMIV” mentioned above).Multiple-choice questions may also have associated weights, if desired.

[0082] Date-type questions can have different levels of precision:MMDDYYYY, MMYYYY, or YYYY. A “date of birth” question would hope to havean MMDDYYYY answer, where a “year of divorce” question only needs a YYYYanswer.

[0083] Inherited questions are those which are, usually, some staticdata elements such as sex, address, race, etc. which are maintained inthe individual's “master file”. If an assessment instrument has, forexample, the “address” question, it can be inherited from the masterfile, eliminating the need to re-key it into the database. Inheritedquestions may or may not be editable. Editable questions can beoverridden, whereas non-editable questions cannot.

[0084] Freeform questions are those questions which permit anyinformation to be entered. There are no answer restrictions in thiscase. These questions are usually used for names or descriptions ofthings that will not be readily analyzed at an aggregate level.

[0085] Each question that is defined contains a prompt that is to bedisplayed on the Assessment Answer Screen (FIGS. 36b, 36 c and 37 c)when the target question is presented for answering. A question can bedesignated as requiring an answer or not.

[0086]FIG. 26 shows the initial Question Maintenance screen. This screendisplays a summary of questions currently in the question repository. Italso shows the question type. A new question may be added to therepository by depressing the Add button, or an existing question may beedited by pressing the Edit button.

[0087] If the Add button is depressed, a Question/Answer MaintenanceScreen (FIG. 27) is displayed. The Question text box 2701 permits typingin a new question. The Question Type combo box 2702 allows selection ofone of the question types. Depending upon the chosen question type, theQuestion/Answer Maintenance Screen (FIG. 27) displays additionalcontrols which further define the answer choices. In all cases, thequestion answer prompt, Description of Choices 2703, is designed tocapture a user-intended prompt.

[0088]FIGS. 28a, 28 b, and 28 c show the Question/Answer MaintenanceScreen as it appears after the Question Type combo box 2702 has selectedquestion types of: Multiple Choice Answers in Answer List 2704, MultipleChoice Answers in Table 2705, and Range (answer must lie withinboundaries) 2706, respectively.

[0089]FIG. 28a shows an Answer List subform where answers can be chosenfrom the answer repository using the answer-selection combo box 2801.FIG. 28b shows the Select Code Table combo box 2822 where one of theimported code tables can be selected: FIG. 28c shows From text box 2833and To text box 2834 where the (inclusive) lower and upper bounds to therange-type (numeric) question are inputted.

[0090] Sub-process Maintain Assessment Instruments 313 is the part ofthe Assessment Instrument Maintenance Module where the AssessmentInstruments 870 (FIG. 8) are maintained.

[0091] An assessment instrument, as defined earlier, is a form orquestionnaire which contains questions. Some of the questions areindependent of one another, and some are related to other questions. Toaccommodate this question-dependency, the concept of Instrument QuestionGroup 910 (FIG. 9) is introduced. An Instrument Question Group 910contains questions which are related, and the “relatedness” is importantin the analysis of the answers. For example, a question group might be“History of Psych Evaluations”. It contains a set of 4 questions: Dateof assessment, diagnosis, treatment required? (Y/N), and date treatmentcompleted. These questions are all related to a historical event, a“psychiatric evaluation.” See FIG. 29a to see the screen where thisquestion group is defined. The highlighted question group 2911 (topsection of screen 2910) has the associated questions (bottom section ofscreen 2920). This is an example of a multi-question (or multi-part)question group.

[0092] Most question groups contain a single question, and are thussingle-part (single-question) question groups. Most instruments containsingle-part question groups.

[0093] To define an assessment instrument using the screen displayed inFIG. 29a, simply depress the Add Instrument 2901 button. Next, addquestion groups using the Add Instrument-Question Group 2912 button. Ascreen depicted in FIG. 30a asks whether the new question group is aSingle-Part 3001 or Multi-Part 3002 question group. Also requested arethe group Sequence # 3003 and a Multiple-Responses Allowed? flag 3004.

[0094] If multi-part question group is chosen, a subform appears as seenin FIG. 30b which requests a Description of the multi-part questiongroup 3005 and the Selection of up to 5 questions 3006. Each question isto be selected from the set of questions in the question repostorymaintained in Maintain Questions 312. The limit of 5 questions per groupis purely an interface issue and is not otherwise restricted by thedatabase design. It can be contemplated that unlimited-question questiongroups can as easily be defined and maintained with some slightadjustment to the relevant screens.

[0095] Once the question group is saved, the Instrument Maintenancescreen (FIG. 29a) returns. At this point, additional question groups maybe added (Add Instrument-Question Group 2912) or deleted (DeleteInstrument-Question Group 2913). Question groups can also be edited byadding (Add Instrument Question 2922) or deleting (Delete InstrumentQuestion 2923) questions in the lower part of the Instrument Maintenancescreen 2920. If Add Instrument Question 2922 is depressed, a new formappears as seen in FIG. 30c.

[0096] Another attribute of an Assessment Instrument 870 (FIG. 8) iswhether or not it is an aggregate-score instrument (see Aggregate Score?checkbox 2902 in FIG. 29a). Such an instrument must have onlysingle-part question groups containing only questions that are of Range2706 (FIG. 28c) question type. Since each Range-type question can onlyhave numeric answers, and each question group has only one question,then each question group inherits the numeric “score” from its only“child” question. The assessment instrument, then, can have a cumulativeor aggregate score which is the sum of the scores of its questiongroups.

[0097] See FIG. 31a. If an Assessment Instrument 870 (FIG. 8) qualifiesaccording to the above criteria and is designated as an aggregate-scoreinstrument, then it can have interpretations associated with itsaggregate scores. If the Aggregate Score? check box 3102 is selected, aScore Intervals button 310 becomes visible. Depressing this buttonbrings up the dialog box 3110 (FIG. 31b). This dialog box permits entryof score intervals {Minimum Value 3111, Maximum Value 3112}, withassociated Interpretation Text 3115. This screen effectively allows theaggregate score to be interpreted. This will be important when viewingassessment summary information in process Record ParticipationInformation 240 (FIG. 3d and 35) and in process Collect and AggregateAssessment Information 1210 (FIG. 12). The score intervals shown in FIG.31b correspond to those defined in the sample Assessment Instrument 870of FIG. 47c.

[0098] Once an Assessment Instrument 870 (FIG. 8) is defined in theAssessment-Instrument Maintenance Module, it may then be used to conductan assessment (FIG. 35c shows a dialog box requesting the selection of apre-defined Assessment Instrument 870). The process of conducting anassessment is described as sub-process Conduct Assessment 342 (FIG. 3d)of the process Record Participation Information 240 (FIG. 2) and will bediscussed at length later.

[0099] The Program Maintenance Module

[0100]FIG. 3b is a decomposition of the Maintain Program Informationprocess 220. The seven sub-processes represented by 321 through 327describe a user interface that maintains program information. Thecorresponding data model focusing on Program Offerings 520 is shown inFIG. 7. These processes are thus briefly summarized below.

[0101] The purpose of the Maintain Program Attributes 321 interface isto provide a means of maintaining basic program information such as theprogram name, its mission, objectives, etc. Agencies play several rolesrelative to the operation of programs. The two primary roles are thoseof providers and funders. A particular agency may provide both of theseroles simultaneously. Defining the basic attributes (e.g. name, address,employees, contacts, etc.) of any agency involved with programs is theobject of Maintain Agency Information 322 process. If the agency is aprovider, the Maintain Agency Provider Information 323 process is wherethis designation is defined. Programs that the agency provides could bedefined in this process, but the preferred embodiment has chosen tomaintain the many-to-many Agency Provider-to-Program relationshipthrough the Maintain Program Offering Information 325 process.

[0102] Maintain Individual Provider Information 324 is where informationabout individuals who work in some capacity in a program's operation(also known as “workers”) is captured. Name and contact information,service role, employing agency, etc. is defined.

[0103] Maintain Program Offering Information 325 is a sub-process ofMaintain Program Information 220 process. One of the main business rulesembodied in the present invention is: agency providers can provide manyprograms, and a program can be provided by many agency providers. Themany-to-many relationship between agency providers and programs createsthe need for a relationship (or entity) to decompose the many-to-manyrelationship into two one-to-many relationships. This new entity isreferred to as Program Offering 520 (FIG. 5). A Program 510 (FIG. 5) canhave one or more Program Offerings 520. A Program Offering 520 has oneProgram 510 and one Agency Provider 540 (FIG. 5). An Agency Provider 540can have one or more Program Offerings 520. See FIGS. 5 and 7 foradditional detail.

[0104] Most “program” attributes are associated with Program Offering520 instead of Program 510, because they can vary by offering. Some ofthese important attributes (aside from the associated Program and AgencyProvider) are: dates of operation, contract info, budget info, fundinginfo, workers, eligibility requirements, referral and other procedures,services offered, location of program offering. These are all maintainedin the Maintain Program Offering Information 325 process.

[0105] Maintain Funding Information 326 is where funding information ismaintained. This includes funds, funding accounts, associated funders,funding requirements and designations, funding amounts and purpose, etc.

[0106] A number of “domain”-type data items are needed to support theprogram offering and program definition. These include repositories of:program components (also known as services; interventions are consideredprogram components as well), termination reasons, accounting codes,worker service roles, etc. The maintenance of this domain information isthe object of process Maintain Other Program-related Information 327.

[0107] The seven sub-processes contribute to what is conceptuallydefined as data store Program Information 320 in FIG. 3b. Thisinformation is physically stored however, in the “Youth” implementationof the preferred embodiment (see FIG. 11), in many tables (e.g.tblProgram, tblAgency, tblProvider, tblProgramOffering, tblFund,domPgmComponents, domTermReasons, etc.) These tables align, notsurprisingly, with the entities described in the Program Offering 520data model of FIG. 7.

[0108] The Participant Maintenance Module

[0109]FIG. 3c is a slightly more detailed description of the MaintainParticipant Information process 230. Like the Maintain ProgramInformation process 220, it is an insignificant component of the presentinvention. The data model Individual Participant/Household (FIG. 6),contains the assumed business rules related to Individuals 630,Individual Participants 530, and their Living Situation 670. The userinterface describing how this participant information is maintained isnot significant to this invention.

[0110] The information maintained by this process is reflectedconceptually in a single data store Participant Information 330, but, inthe “Youth” implementation of the preferred embodiment, several tablesare used to contain this information: tblYouth (the participant),tblFamMem, tblFamMemName, tblAddress, domLivingSituation,domParentalStatus, etc.

[0111] It can be contemplated that any number of tables could be used torepresent Participant Information 330. For the purposes of thisillustration, only two are used: tblYouth and tblFamMem, and assume thata single name, single address, and otherwise stable individualattributes reside in one of these two tables. See FIGS. 10 and 11 to seethese relationships.

[0112] Note: The tblAddress and tblFamMemName were used to storemultiples addresses and names, respectively, because the programenvironment was one in which the history of the individual's address(and alias names) was important in tracking the individual'sparticipation. The choice to use these “hard-coded” additional tablescould have been circumvented by using an assessment instrument withquestions of: “addresses” and “names.”

[0113] In general, assessment instruments are useful for capturingmultiple longitudinal snapshots of any needed information. Anyhistorical data can be easily maintained through the assessmentactivity. For example, a single- (or few-) question assessmentinstrument (e.g. “Address”) could be developed to capture one or moredata items (e.g. “Street Address”, “City”, “State”).

[0114] In general, a good rule of thumb might be that the staticcharacteristics of an individual (Sex, Race, DOB, SSN) or somewhatstatic characteristics which are fundamental (Name, Address) yet have noneed of being tracked historically, can be attributes of the IndividualParticipant 530. Otherwise, it is useful to capture IndividualParticipant's characteristics via an Answered Question 925 during anIndividual Assessment Conducted 570 (snapshot).

[0115] Record Participation Module

[0116] Though the focus of the Record Participation Module is on theprocess Record Participation Information 240, as depicted in FIG. 3d, itis helpful to refer to the corresponding section of the data model (FIG.8, shown generally by numeral 800) that focuses on IndividualParticipation Activity 560. Frequent reference will be made to objectsin FIG. 8.

[0117]FIG. 3d, shown generally by numeral 240, is a decomposition of theRecord Participation Information process 240 into eight sub-processes341 through 348. These are: Initiate Program Participation 341, ConductAssessment 342, Log Worker Assignment Information 343, ContactInformation 344, Log Service Receipt/Completion Information 345, Log LogIncident Information 346, Log Other Activity/Event Information 347, andTerminate Program Participation 348, respectively.

[0118] These processes maintain information in the data storeParticipation Information 340 (a subset of data store Database 12);information which relates to an Individual Participant's 530Participation 550 in a Program Offering 520.

[0119] Note that data stores Program Information 320 and ParticipantInformation 330 are used as input to all of the eight sub-processes. Thedata store Assessment-Instrument Definition Information 310, however, isused only for the particular sub-process Conduct Assessment 342. Theinformation in the data store Assessment-Instrument DefinitionInformation 310 provides the questions to be asked during the ConductAssessment process 342.

[0120] In the specific implementation of the preferred embodiment, asdocumented in FIGS. 11 (and higher), the Individual Participant 530referred to in FIG. 10 (and lower) is specifically referred to as“Youth.” FIG. 11 uses specific table and field names to implement thegeneral entities and attributes, respectively, that are found in FIG.10. For example, Individual Participant 530 of FIG. 10 corresponds totblYouth 530′ of FIG. 11. And Participation 550 of FIG. 10 correspondsto tblYouthPgmPartic 550′ of FIG. 11. Note the convention of using the“prime” notation to designate the embodiment of a general entity as aphysical implementation of that entity, in this case, a “table.”

[0121] In the preferred embodiment, the process of creating aParticipation relationship 550 between an Individual Participant 530 anda Program Offering 520 is created from the Individual Participant's 530side, i.e. from the “Youth's” side, vs. the Program Offering's 520 side.

[0122] For example, the Youth maintenance screen depicted in FIG. 32 andshown generally by numeral 3200, contains a Program Participation button3201. Depressing this button enables Participations 550 to becreated/edited/terminated for the particular youth whose record isdisplayed on the Youth maintenance screen 3200.

[0123] It can be contemplated that the Participation 550 records couldbe created/edited or deleted, as well, from the Program Offering's 520side. In fact, many “program roster” type reports present this “view” ofthe many-to-many relationship Participation 550 that exists betweenIndividual Participant 530 and Program Offering 520.

[0124] In the “Youth” implementation of the preferred embodiment, asdescribed above, the Program Participation button 3201 on the Youthmaintenance screen 3200 is the gateway to the Record ParticipationInformation process 240, i.e. the means of initiating each of the eightprocesses (341 through 348) shown in FIG. 3d.

[0125] Each of the eight processes is a user-interface that maintainsinformation in the data store Participation Information 340. Associatedexemplary screens drawn from the “Youth” implementation of the preferredembodiment will be used to describe these eight processes. The samplescreens are invoked from the Youth Program Participation screendisplayed in FIG. 33a, shown generally by numeral 3300. And the YouthProgram Participation screen 3300 is invoked from the ProgramParticipation button 3201 of the Youth maintenance screen 3200.

[0126] Youth Program Participation Screen

[0127] The Youth Program Participation screen 3300 shows a summarylisting of all of the Program Offerings 320 in which the target youth,whose name is shown in the Name text boxes 3310, is currentlyparticipating or has participated. This list of Participations 550 isshown in the Participation Summary subform 3320. The left-most boxes ofthe Participation 550 records, referred to as Participation RecordSelectors 3325, are used to select a particular Participation 550. Thebottom left section of the screen, the Program Participation Activitysection 3340, contains a number of buttons—3342 through 3348—whichinvoke the processes 342 through 348, respectively, shown in FIG. 3d.For example, the Workers button 3343 invokes the Log Worker AssignmentInformation process 343. And the Log Worker Assignment Informationprocess 343 is responsible for capturing the Worker Associated activity820 (FIG. 8).

[0128] In the upper right-hand corner of the Youth Program Participationscreen 3300 is another button, Program Initiation/Activity 3341 . Thisbutton invokes the Initiate Program Participation process 341 of FIG.3d. Finally, two buttons in the Reports section of the screen, PgmActivity Summary 3350 and Assessment Summary 3360, invoke two summaryreports.

[0129] Initiate Program Participation Process

[0130] The New Program Activity screen, depicted in FIG. 34a and showngenerally by numeral 3400, is the user interface associated with theprocess Initiate Program Participation 341. Depressing the ProgramInitiation/Activity button 3341 on the Youth Program Participationscreen 3300 opens this New Program Activity screen 3400.

[0131] The New Program Activity screen 3400 permits the selection of aProgram Offering 520 via the Select Program Offering combo box 3401. NewActivity Type 3402 combo box provides the means to select a particularActivity Type 580. In this preferred embodiment, an assumption is madethat only pre-initiation- and initiation Activity Types 580 areavailable for selection. Pre-initiation activities include activitieslike “was referred to”, “was accepted into” and “was denied acceptanceinto.” The initiation Activity Type 580 “PARTICIPATION BEGAN” creates anew Participation 550 record (i.e. a new record in the tblYouthPgmPartictable 550′). Multiple pre-initiation activities may be logged, eachhaving an activity date. An Activity list box 3405 displays thesepre-initiation activities. Importantly, the date of the activity must beentered into the Date of Activity text box 3404.

[0132] Once the initiation activity (i.e. Activity Type580=“PARTICIPATION BEGAN”) is posted (i.e. the OK button is depressed),control returns to the Youth Program Participation screen 3450 shown inFIG. 34b. This screen 3450 differs from 3300 in that it contains a newParticipation 550 record 3460 in its Participation Summary subform 3470.

[0133] Once a Participation 550 is created by the Initiate ProgramParticipation process 341, an Individual Participation Activity 560 canbe associated with that Participation 550.

[0134] Individual Participation Activities

[0135] To log Individual Participation Activities 560, first, select theappropriate Participation 550 record using the Participation SummaryRecord Selector 3320 on the Youth Program Participation screen 3300.Then depress one of the buttons in the Program Participation Activitysection 3340. In the specific implementation of the preferredembodiment, there are five Major Activity 860 (also known as MajorEvent) buttons—Assessments 3342, Workers 3343, Contacts 3344, Components3345, and Incidents 3346—and a Miscellaneous Activity button 3347 to logMinor Activities 850.

[0136]FIG. 8 depicts the five Major Activities 860—Individual AssessmentConducted 570, Worker Associated 820, Contact Made 875, Service Received810 and Incident Occurred 845—corresponding to the five Major Activity860 buttons: 3342 through 3346 on the Youth Program Participation screen3300, respectively. These five Major Activities 860 are those requestedby the client for whom the “Youth” implementation of the preferredembodiment was developed. It is contemplated that other Major Activities860 could be readily defined as well. Some examples of additional MajorActivities 860 are shown in FIG. 8 as: Service Plan Developed 830,Referral Made 840, Award Received 855 and Goods Received 865.

[0137] The difference between Major Activities 860 and Minor Activities850 relates to the number and quality of activity-specific attributesthat need to be stored and reported on, and thus the need foractivity-specific database structures to hold those data items. EachMajor Activity 860 has a special table (e.g. tblYouthPgmParticAsst,tblYouthPgmParticWorker, tblYouthPgmParticContact, etc.) to hold dataitems which further qualify the activity (beyond Activity Type 580 anddate of activity). Expanding/extending the set of five Major Activities860 beyond those found in the “Youth” implementation of the preferredembodiment would require only minor database and functionalmodifications.

[0138] Minor Activities 850 can be defined by the user in aMiscellaneous-Data interface in the “Youth” implementation of thepreferred embodiment. In that system, Activity Types 580 are stored inthe table domActivityType. Once an activity type is recorded in thedomActivityType table, it can then be selected as a Minor Activity 850.The procedure for logging Minor Activities 850 will be discussed afterthe processes for logging Major Activities 860 are described.

[0139] Logging Major Activities

[0140] The most complex Major Activity 860 type is that of IndividualAssessment Conducted 570 (in FIG. 3d). Thus the process for logging thisActivity Type 580, Conduct Assessment 342 is discussed first.

[0141] Conducting an Assessment

[0142] The process Conduct Assessment 342 is invoked from the YouthProgram Participation screen 3300 by, first, selecting the targetParticipation 550 (using the Participation Record Selector 3325 in FIG.33a) and then, pressing the Assessment button 3342. The resulting YouthProgram Participation-Assessments screen, depicted in FIG. 35a and showngenerally as numeral 3500, will open.

[0143] This Youth Program Participation-Assessments screen 3500 showsall of the Individual Assessments Conducted 570 for the targetParticipation 550 in an Assessment Summary list box 3530. For eachIndividual Assessment Conducted 570, the following data items are shown:Interview Date 3531, Assessment Instrument name 3532, Caseworker incharge of the interview/assessment 3533, and, if relevant, an AggregateScore 3534, with corresponding Score Interpretation 3535. An AssessmentSummary report 3545, invoked by the Assessment Summary button 3540, isshown in FIG. 35b.

[0144] The Add 3550, View/Edit 3560, and Delete 3570 buttons on screen3500 are used to add, view/edit, or delete an Individual AssessmentsConducted 570 activity, respectively. Depressing the Add button 3550,opens the Add New Assessment screen depicted in FIG. 35c and showngenerally as numeral 3555. Screen 3555 contains an Instrument combo box3556, from which an Assessment Instrument 870 must be selected. Alsorequested are the caseworker and interview date (i.e. date ofassessment). Posting the information creates an Individual AssessmentConducted 570 activity, implemented in the “Youth” implementation of thepreferred embodiment, as a new record in tblYouthPgmParticAsst 570′ (inFIG. 11). Also, as with all Individual Participation Activities 560, anew record is also added to tblYouthPgmActivity 560′ (FIG. 11). TheYouth Program Participation-Assessments screen 3500 is again displayed,now showing a new Assessment record in the Assessment Summary list box3530.

[0145] Answering Assessment Questions

[0146] The View/Edit 3560 button is used to open the Conduct AssessmentInterview screen found in FIG. 36a and shown generally as numeral 3600.Beneath the header section, which contains the name of the AssessmentInstrument 870, the Youth 530′ name, the interview date and thecaseworker name, there are three additional sections: a Question Groupsection 3610, a Question section 3620, and an Answer section 3630.

[0147] Sections Question Group 3610 and Question 3620 are linked inthat, for the selected Instrument Question Group 910 (FIG. 9) in theQuestion Group section 3610 (noted by the presence of a triangle in theQuestion Group Record Selector 3611), the associated InstrumentQuestion(s) 920 is (are) displayed in the Question section 3620. Thisassociation between an Instrument Question Group 910 and InstrumentQuestions 920 was defined in Maintain Assessment Instrument process 313(FIG. 3a) described earlier. This relationship is also depicted in FIG.9.

[0148] In the Question Group section 3610, there are two buttons to theright of the list of Instrument Question Groups 910—an Add Answer button3612 and a Delete Answer button 3613. These buttons are visible onlywhen relevant—e.g. the Delete Answer button 3613 is not visible if thereis no answer yet supplied.

[0149] In the Question section 3620, there is an Answer/Edit Questionbutton 3622 on the right-hand side of the screen. This button providesthe means to define Answer Sets 3639. For a given Instrument QuestionGroup 910 having a set of associated Instrument Questions 920, asingle-response set of answers for each of those questions is called anAnswer Set.

[0150] In the preferred embodiment, the relationship between theQuestion section 3620 and the Answer section 3630 is somewhat differentthan the relationship between the Question Group section 3610 and theQuestion section 3620. For each Instrument Question 920 (with index #'sfrom 1 up to 5, determined from the relative Instrument Question 920sequence #), there is a hard-programmed column in the Answer section3630 which corresponds to the target Instrument Question 920 (asselected by the Instrument Question Record Selector 3621). For example,the Instrument Question 920 which has an index # of 3 (i.e. is the3^(rd) Instrument Question 920 corresponding to the target InstrumentQuestion Group 910), will have its corresponding Answer 960 in the3^(rd) position in the Answer Set 3639 of the Answer section 3630.

[0151] Each row in the Answer section 3630 corresponds to an Answer Set3639. The presence of multiple rows implies that there are multipleAnswer Sets 3639 associated with the target Instrument Question Group910 (i.e. as selected by the Question Group Record Selector 3611). AnInstrument Question Group 910 is eligible to have multiple Answer Setsif its Multi-Response checkbox 3614 is checked. Otherwise, theInstrument Question Group 910 can only have a single Answer Set 3639(i.e. a single row of answers)..

[0152] The particular interface was designed to record multipleinstances of multi-part answers to multi-part questions in a manner thatwas intuitive to a user, that would require as few keystrokes aspossible, and would display an appropriate amount of data on a singlescreen. It is contemplated that improved interfaces can be readilydeveloped that would implement the flexible assessment data model, yetbe more user-friendly. The development of multiple interfaces (e.g. onefor single-response/single-part instruments, another forsingle-response/multi-part instruments etc.) may be an attractiveapproach.

[0153] The Conduct Assessment process 342 will be illustrated by twoexamples using exemplary screens from the “Youth” implementation of thepreferred embodiment. The first example, (I), assumes an AssessmentInstrument 870 which is NOT an Aggregate-Score instrument and containsat least one Multi-Response Instrument Question Group 910. It isillustrated by the screens shown in FIGS. 36a, b, c and d. This exampleis further divided into two scenarios: a) a multi-response andmulti-part scenario, and b) a single-response and single-part scenario.The second example,(II), assumes an Assessment Instrument 870 which isan Aggregate Score instrument containing no Multi-Response InstrumentQuestion Groups 910. Furthermore, each Instrument Question Group 910 isa Single-Part (i.e. has only one associated Instrument Question 920.)This example is reflected in screens shown in FIGS. 37a and b.

EXAMPLE I Non-Aggregate-Score Assessment Instrument

[0154] a) Multi-Response/Multi-Part Instrument Question Group “Historyof Psych Eval Assessments”

[0155] 1) Refer to screen 3600 in FIG. 36a. In the Question Groupsection 3610, select the Instrument Question Group 910 named “History ofPsych Eval Assessments” (Note that this Instrument Question Group's 910Multi-Response checkbox 3614 is checked. It can thus have multipleresponses, i.e. multiple Answer Sets 3639. This Instrument QuestionGroup 910 is also associated with multiple Instrument Questions 920,shown in the Question section 3620.

[0156] 2) Depress the Add Answer button 3612 (to add the first “empty”Answer Set 3639).

[0157] 3) Next, in the Question section 3620, select the desiredInstrument Question 920 to be answered.

[0158] 4) Select, for example, the “History of Psych EvalAssessments-Diagnosis”, the 2^(nd) Instrument Question 920, as definedby its relative sequence #.

[0159] 5) Now depress the Answer/Edit Question button 3622. This actionwill open the Answer Question dialog box illustrated in FIG. 36b andshown generally by numeral 3650.

[0160] 6) The Answer control 3652 in the AnswerQuestion dialog box 3650is customized to handle the Question Type 931 associated with theselected Instrument Question 920. In this example, the Answer control3652 is a combo box whose record source is the table whose name,“domDSMIV”, was pre-specified in the Maintain Questions process 312, asthe “domain” of answers for the Instrument Question 920=“History ofPsych Eval Assessments-Diagnosis.”

[0161] 7) In addition to the Answer control 3652, the dialog box 3650also contains three read-only text boxes whose values have beenpre-defined (in the Maintain Question process 312 described earlier.)The first, labeled Question 3651, contains the question description“History of Psych Eval Assessments-Diagnosis.” The second read-only textbox, labeled Choices 3653, displays the question prompt “DSM-IV code.”(This too, was pre-defined in the Maintain Question process 312.)Finally, the last read-only text box, shown in the lower part of thedialog box and numbered 3654, displays the Question Type 931. In thisexample, the Question Type 931 is “Multiple Choice Answers in Table.”(See FIG. 27b, the screen that assigned this Question Type 931 to theQuestion 930=“History of Psych Eval Assessments-Diagnosis.”

[0162] 8) After selecting a particular diagnosis code, in this example,“Acute Stress Disorder”, depress the OK button to save the answer as the2^(nd) component of the 4-component Answer Set 3639. In particular, the“Acute Stress Disorder” is saved in the control 3632.

[0163] 9) Repeat the “answer question” process for each of the fourInstrument Questions 920 associated with the “History of Psych EvalAssessments” Instrument Question Group 910.

[0164] 10) To add a second response to this multi-response InstrumentQuestion Group 910, depress the Add Answer button 3612 a second time.This will add a second “empty” Answer Set 3639. Answer each of the fourInstrument Questions 920.

[0165] 11) The Conduct Assessment Interview screen displayed in FIG. 36cand shown generally by numeral 3660, shows the resulting answers to the“History of Psych Eval Assessments” Instrument Question Group 910.

[0166] b) Single-Response/Single-Part Instrument Question Group “CurrentSchool Attendance . . . ”

[0167] 1) Refer to screen 3670 in FIG. 36d. In the Question Groupsection 3671, select the Instrument Question Group 910 named “CurrentSchool Attendance Status . . . ” (Note that this Instrument QuestionGroup's 910 Multi-Response checkbox 3674 is NOT checked. It can thushave one response, i.e. one Answer Set 3679. This Instrument QuestionGroup 910 is associated with a single Instrument Question 920 having thesame name as the Instrument Question Group 910 designated by 3675, i.e.“Current School Attendance . . . ”)

[0168] 2) Depress the Add Answer button 3672 (to add an “empty” AnswerSet 3679). Next, in the Question section 3677, select the desiredInstrument Question 920 to be answered. Since this is a Single-PartInstrument Question Group 910, there is no need to select an InstrumentQuestion 920. There is only one question, and it is “Current SchoolAttendance Status (less than 10 is Satisfactory).” By default, it is“selected.”

[0169] 3) Depress the Answer/Edit Question button 3672. This action willopen the Answer Question dialog box illustrated in FIG. 36e, and showngenerally by numeral 3680.

[0170] 4) The Answer control 3682 in the AnswerQuestion dialog box 3680is customized to handle the particular Question Type 931 associated withthe selected Instrument Question 920. In this example, the Answercontrol 3682 is a combo box whose record source is the set of answers(i.e. the Answer Repository) pre-defined in the Maintain Questionsprocess 312.

[0171] 5) In addition to the Answer control 3682, the dialog box 3680also contains three read-only text boxes whose values have beenpre-defined (in the Maintain Question 312 process described earlier.)The first, labeled Question 3681, contains the question description, inthis case “Current School Attendance Status (less than 10 isSatisfactory).” The second read-only text box, labeled Choices 3683,displays the question prompt, in this case:“Satisfactory/Unsatisfactory/Non Applicable (i.e. not in school).” (Thistoo, was pre-defined in the Maintain Question 312 process.) Finally, thelast read-only text box, shown in the lower part of the dialog box andnumbered 3684, displays the Question Type 931. In this example, theQuestion Type 931 is “Multiple Choice Answers in Answer List.” (See FIG.27a, the screen that assigned this Question Type 931 to the Question930=“Current School Attendance . . . ”

EXAMPLE II Aggregate-Score Assessment Instrument withSingle-Response/Single-Part Instrument Question Groups

[0172] Refer to screen 3700 in FIG. 37a.

[0173] 1) In the Question Group section 3710, select the InstrumentQuestion Group 910 named “LOSI . . . Substance Abuse” This InstrumentQuestion Group 910 is associated with a single Instrument Question 920having the same name as the selected Instrument Question Group 910, i.e.“LOSI Substance Abuse”.

[0174] 2) Depress the Add Answer button 3712 (to add an “empty” AnswerSet 3739). Next, in the Question section 3720, select the desiredInstrument Question 920 to be answered (identified by the InstrumentQuestion Record Selector 3765). Since this is a Single-Part InstrumentQuestion Group 910, there is no need to select an Instrument Question920. There is only one Instrument Question 920, and it is “LOSISubstance Abuse”, with a Question Type 931=“Range (answer must liewithin range boundaries)”. By default, it is “selected.”

[0175] 3) Depress the Answer/Edit Question button 3722. This action willopen the Answer Question dialog box illustrated in FIG. 37b, and showngenerally by numeral 3750. The Answer control 3752 in the AnswerQuestion dialog box 3750 is customized to handle the particular QuestionType 931 associated with the selected Instrument Question 920. In thisexample, the Answer control 3752 is a text box having validationprocedures which ensure that the numeric answer supplied does indeedfall between the two range boundaries, inclusive.

[0176] 4) In addition to the Answer control 3752, the dialog box 3750also contains three read-only text boxes whose values have beenpre-defined (in the Maintain Question process 312 described earlier.)The first, labeled Question 3751, contains the question description, inthis case “LOSI Substance Abuse.” The second read-only text box, labeledChoices 3753, displays the question prompt “From 0 to 5 (Low=0;Moderate=1 to 2; High=3 to 5)” (This, too, was pre-defined in theMaintain Question process 312.) Finally, the last read-only text box,shown in the lower part of the dialog box and numbered 3754, displaysthe Question Type 931. In this example, the Question Type 931=“Range(answer must lie within range boundaries)”. (See FIG. 27c, the screenthat assigned this Question Type 931 to the “LOCI Substance Abuse”Question 930.

[0177] Deleting Assessment Answers

[0178] Deleting the answers for an Instrument Question Group 910 isaccomplished from the Conduct Assessment Interview screen 3600 by,first, selecting the Instrument Question Group 3611, and then depressingthe Delete Answer button 3613. This will delete all Answer Sets in theAnswer section 3630 for the selected Instrument Question Group 910.

[0179] When the Assessment Interview is completed, depress the exitbutton to return to Youth Program Participation-Assessments screen 3500.An Assessment record for the Assessment just conducted should be visiblein the Assessment Summary subform 3550.

[0180] Logging Other Major Activities

[0181] The second Major Activity of Record Participation Information 240is that of Log Worker Assignment Information 343. The user interface forthis process is shown by the exemplary Youth ProgramParticipation-Workers screen displayed in FIG. 38a, and generally shownby numeral 3800. This screen is invoked from the Youth ProgramParticipation screen 3300, by, first selecting a particularParticipation 550, and then, by pressing the Workers button 3343. Thisscreen shows, for the Participation 550 selected from screen 3300, theset of associated workers. These workers are listed, one per record, inthe Worker List section 3810 of screen 3800, and correspond to WorkerAssociation 820 activities. For each Worker Associated 820, a recordshows the worker name, start date, end date, service role, termination,phone number and notes. It can be contemplated that additionalworker-specific attributes could be added to those listed above.

[0182] To add a new Worker Association 820 activity, it is necessary todepress the Add button 3820. This will add new records to the generictblYouthPgmActivity 560′ table and to the activity-specifictblYouthPgmParticWorker table 820′.

[0183] A second View/Edit Program Workers screen, depicted by FIG. 38band shown generally by numeral 3850, displays some additional detailabout the selected worker and provides a better interface for entry ofnotes. This screen is invoked from the Youth ProgramParticipation-Workers screen 3800, by, first selecting a particularWorker Association record 820 from the Worker List section 3810, andthen depressing the View/Edit button 3830. Though there is littleadditional information on screen 3850 than that shown in screen 3800, itcan be contemplated that, when more Worker Association 820 attributesare required, the need to have a second “detailed” screen such as screen3850 may be greater.

[0184] Depending upon the business rules required, the set of workersfrom which to choose when establishing a Worker Association 820, may, ormay not be filtered by, say, a rule requiring the worker to be employedby the Agency which is the Agency Provider of the Program of the ProgramOffering 520 of the Participation 550.

[0185] The third Major Activity of Record Participation Information 240is that of Log Contact Information 345. The user interface for thisprocess is shown by the exemplary Youth Program Participation-Contactsscreen displayed in FIG. 39a, and generally shown by numeral 3900. Thisscreen is invoked from the Youth Program Participation screen 3300, byfirst selecting a particular Participation 550, and then, by pressingthe Contacts button 3344. This screen shows, for the Participation 550selected from screen 3300, the set of contacts made.

[0186] The general functioning of the Log Contact Information process344 is similar to that described for the Log Worker Information process343 above. There are two screens, a summary-listing-level screen, inthis case, screen 3900, and a detailed single-record screen, in thiscase, screen 3950 (shown in FIG. 39b).

[0187] The fourth Major Activity of Record Participation Information 240is that of Log Service Receipt/Completion Information 345. The generalfunctioning of the Log Service Receipt/Completion Information process345 is similar to that described for the Log Worker Information process343 above. There are two screens, a summary-listing-level screen, inthis case, screen 4000, and a detailed single-record screen, in thiscase is not shown.

[0188] One important note to make about the Program Component selectioncombo box is that the record source of the combo box contains only thoserecords (i.e. program components) which have previously been defined as“offered” by the Program Offering 820 in the Maintain ProgramInformation process.

[0189] The fifth Major Activity of Record Participation Information 240is that of Log Incident Information 346. The general functioning of theLog Incident Information process 346 is similar to that described forthe Log Worker Information process 343 above. There are two screens, asummary-listing-level screen, in this case, screen 4100, and a detailedsingle-record screen, in this case, 4150.

[0190] Minor Activities

[0191] Minor Activities 850 are logged by depressing the MiscellaneousActivity button 3349 on the Youth Program Participation screen 3300. TheProgram Activity screen illustrated in FIG. 42a and generally shown asnumeral 4200 appears. It shows a summary of all activity logged to date(Activity Summary list box 4201) for the Youth 530′ whose name appearsin the upper right hand corner of the screen 4202 relative to theParticipation 550 in the Program Offering 520 shown in the read-onlyProgram Name text box 4203. It also provides the opportunity to logadditional Individual Participation Activity 560 using the New ActivityType combo box 4204 and the Date of Activity text box 4205. ActivityNotes 4206 can be captured as well. The Program Activity screen 4200 isvery similar to the New Program Activity screen 3400.

[0192] A report-version of the activity-summary information found on theProgram Activity screen 4200 is shown in FIG. 42b, generally shown bynumeral 4250. It is invoked from the PgmActivity Summary button 3320 ofthe Youth Program Participation screen 3300.

[0193] Program Participation Termination

[0194] Refer again to the Youth Program Participation screen 3300. Whena Participation 550 is to be terminated, it must first be selected byclicking the record selector to the left of the appropriatenon-terminated Participation 550 record. (Note: Non-terminatedParticipation 550 records are those with neither an End Date norTermination Reason.)

[0195] Depressing the Program Termination/Transfer button 3348 brings upthe Program Termination/Transfer Activity screen of FIG. 43, generallyshown by numeral 4300. In the “Youth” implementation of the preferredembodiment, the option to transfer from one Program Offering 520 toanother was accomplished through the Transfer/Terminate buttons 4301,thus abbreviating the process of ending one Program Offering 520 andbeginning another. The termination date and termination reason arerequired input to the Date of Termination 4301 text box and theTermination Reason 4302 combo box. These “boundary” attributes of theParticipation 550 are critical for program-evaluation purposes. Thiswill be illustrated in the Program Evaluation Module which follows.

[0196] Program Evaluation Module

[0197]FIG. 3e, shown generally by numeral 250, shows a decomposition ofthe Prepare and Output Reports process 250 of FIG. 2 into twosub-processes, Prepare & Output Participant Information 351 and Prepare& Output Program Information 352. A sub-process of the latter, namelythe Assess/Evaluate Program Effectiveness process 410, is shown in FIG.4 and is of relevance to the present invention. While theAssess/Evaluate Program Effectiveness process is described inconjunction with the other four modules above, it is contemplated thatthis module can be employed to assess programs based on assessmentinformation collected using processes other than those described above.FIG. 4, shown generally by numeral 400, shows the input data stores tothe process Assess/Evaluate Program Effectiveness 410:Assessment-Instrument Definition Information 310, Program Information320, Participant Information 330 and Participation Information 340. Thetwo most relevant data stores to the present invention are theAssessment-Instrument Definition Information 310 and the ParticipationInformation 340.

[0198] Though the focus of the Program Evaluation Module is on theprocess Assess/Evaluate Program Effectiveness 410, depicted in FIG. 4,and its sub-processes, shown in FIGS. 12 through 18, it is helpful torefer to the corresponding sections of the data model. FIGS. 10 and 11,shown generally by numerals 1000 and 1100, respectively. These segmentsof the overall data model focus on how assessments and programparticipation activity can be used to evaluate program effectiveness.

[0199]FIG. 10 shows the conceptual entities, relationships andattributes most directly involved in generating program-evaluationinformation from Participation 550, Individual Participation Activity560, and Individual Assessments Conducted 570. This data model clearlyshows how the database might be navigated to search for basic programevaluation information. Notice the path between the Program EvaluationConducted 590 and the Individual Assessment Conducted 570. Entities 550,560, 860, 570, 915 and 925 are classified as belonging to the data storeParticipation Information 340. Entities 870, 970, 920 and 930 wouldbelong to the data store Assessment-Instrument Definition Information310.

[0200]FIG. 11 mirrors FIG. 10, but uses specific objects in the “Youth”implementation (namely tables and fields) instead of the more abstractobjects of FIG. 10. The data model segments in FIGS. 19 through 24,shown generally by numerals 1900 through 2400, respectively, describehow queries might navigate through the data model to measure severalexemplary program outcome indicators.

[0201] In addition to the stored information from the four data stores,it is assumed that a user directing the Assess/Evaluation ProgramEffectiveness 410 may want to specify parameters qualifying theanalysis. For example, the user may want to specify which Program 510 orProgram Offering 520 is to be analyzed. The evaluation period shouldalso be specified by the user. Other parameters might be the type ofevaluation (e.g. “compare first and last assessments”, “use only thelast assessment”, “use only assessments conducted 6 months afterbeginning program”, “use only the last assessments conducted aftercompletion of program”, etc.), Individual Participant 530 attributes(e.g. race, sex, etc.) or other factors, Assessment parameters(Assessment Instrument 870 to use, Instrument Question(s) 920 to use,outcome indicators, targets, type of comparison, etc.), and type ofoutput (e.g. report, file, graphic format, summary/detail, etc.) FIGS.44 through 46, shown generally by numeral 4400 through 4600,respectively, show sample screens that solicit these parameters andinvoke some “reports.”

[0202] Program Evaluation Example

[0203] For the purpose of illustrating how the present embodiment couldsupport program evaluation, the following scenario is presented. Thescenario is set in the context of the “Youth” implementation of thepreferred embodiment. It assumes that the database is that depicted indata model 1100 of FIG. 11. (Specific table names from FIG. 11 will bereferenced in the sub-process of Assess/Evaluate Program Effectivenessprocess 410 as they are described in FIGS. 12 through 18.)

[0204] The following evaluation assumptions apply. Assume that aparticular Program Offering 920 is to be evaluated. In the sample, theProgram Name is “Intensive Probation”, and the Program Provider is“Lorain County Domestic Relations Court”. Two program outcome indicatorsare to be used.

[0205] Assume the first indicator is the youth's school attendance. Inthe context of the present invention, this indicator can be measured byexamining answers to the Instrument Question 920 whose description is“Current School Attendance Status (less than 10 is Satisfactory)”.

[0206] The second indicator is a surrogate measure of youth needs andrisk based upon eight questions. The sample Assessment Instrument 870shown in FIGS. 47 (a, b, and c) shows precisely the questions, as wellas the aggregate-score interpretations. The name of this AssessmentInstrument is “LOSI Assessment of Risks and Needs.”

[0207] For youth participating in the Program Offering 920, consideronly those youth which were assessed twice: once at the beginning of theParticipation 550 and once later on. Assume that the date of the secondassessment is not relevant except that it must be conducted after thefirst. Assume also that the second assessment need not occur while theyouth was participating in the Program Offering 920; it could have beenconducted after the Participation 550 ended.

First Indicator Instrument Question-based—“School Attendance”

[0208] For all youth participating in the program, document thefirst-assessment question answer relative to the last-assessmentquestion answer. Present the results in each of the following outputformats:

[0209] 1. Look at the “before and after” answers, including demographicinformation, in tabular form. (See FIG. 44: 4430 and 4440)

[0210] 2. Look at the “before and after” answers, including demographicinformation, in extract file output form. . (See FIG. 44: 4430 and 4450)The purpose of have the extract-file format is to be able to utilizemore sophisticated analysis tools such as SPSS or SAS. The extract filewould be used as input into the analysis tool.

[0211] 3. Look only at the most recent answers by race, in pie-chartform. (See FIG. 45: 4530 and 4560)

Second Indicator Assessment Instrument-Based—“LOSI” Surrogate Score

[0212] For all youth participating in the program, analyze thefirst-assessment scores relative to the last-assessment scores todetermine how those scores changed.

[0213] Present the results in each of the following output formats:

[0214] 1. Look at the “before and after” scores, including demographicinformation, in tabular form. (See FIG. 46: 4630 and 4640)

[0215] 2. Look at the “before and after” scores, including demographicinformation, in extract file output form. (See FIG. 46: 4630 and 4650)

[0216] 3. Look at the change in scores between the first and lastassessments. Show in bar-chart form. (See FIG. 46: 4630 and 4690)

[0217] Assess/Evaluate Program Effectiveness Process Decomposition

[0218]FIG. 12, shown generally by numeral 1200, decomposes theAssess/Evaluate Program Effectiveness into two sub-processes: Collectand Aggregate Assessment Information 1210 and Present Program EvaluationInformation 1220. There are three parameters shown: PgmOffer ID 1201,QuestID 1202 and InstID 1203. In addition, four specific tables areshown as intermediary tables: tblEvalFirstLastAssts 1211,tblEvalAsstAnswers 1212, tblEvalFirstLastAsstsWithAnwersScores 1213 andtblEvalDemogFirstLastAsstsWithAnswersScores 1214. The fields withinthese tables are shown in FIGS. 14 through 19—process decompositions ofthe Collect and Aggregate Assessment Information 1210.

[0219]FIG. 13, shown generally by numeral 1300, decomposes the Collectand Aggregate Assessment Information 1210 from FIG. 12 into fivesub-processes. However, process 1320 and 1330 are mutually exclusive.Depending upon whether the indicator is Question-based orIndicator-based, one of the two Step 2 processes will be used. So, foreach type of indicator measurement (i.e. Question-based orInstrument-based), the entire Collect and Aggregate AssessmentInformation process 1210 requires four steps. FIGS. 48 through 51 showspecific queries that could be used to carry out the 4-step Collect andAggregate Assessment Information process 1210 in the “Youth”implementation of the preferred embodiment.

[0220] If a Question-based indicator is to be measured, Step 2_Quest GetAnswers 1320 is used. If an Instrument-based indicator is used, the Step2_Inst Get Scores and Interpretation 1330 is used. The remaining threeprocesses (1310, 1340 and 1350) are used in measuring both types ofindicators.

[0221] For example, for the first indicator, a Question-based indicator,the four processes: Step 1 Get Paired Assessments 1310, Step 2_Quest GetAnswers 1320, Step 3 Join Assessments to Answers and Scores 1340 andStep 4 Join Answered/Scored Assessments to Demographic Profile Info 1350are used.

[0222] For the second indicator, an Instrument-based indicator, the fourrelevant processes are: 1310, 1330, 1340 and 1350. It can becontemplated that alternative processes (alternate queries or storedprocedures) could be used to navigate through the database to measurethe specified indicators. This is one example.

[0223] Measuring the First Indicator (Instrument Question-based—SchoolAttendance)

[0224] The first step is depicted in FIG. 14. The data flow diagramshows the specific fields within the data stores (tables), that areinput to and output from the process Step 1 Get Paired Assessments 1310.It also shows the parameter PgmOfferID 1201 as an input. The object ofthis process is to find assessments that satisfy the date and pairingcriteria expressed in the evaluation assumptions,—e.g. first and lastassessments occurring after Participations 550 in the specified ProgramOffering 920 (i.e. Intensive Probation/Lorain County Domestic RelationsCourt) begin. This process does not care about the type of assessmentinstrument used, nor of the resulting answers or scores.

[0225] To illustrate the navigation within the database corresponding toprocess Step 1 Get Paired Assessments 1310, refer to the data model 1900shown in FIG. 19.

[0226] The second step, depicted in FIG. 15 and shown generally bynumeral 1500, describes the process Step 2_Quest Get Answers 1320. Itrelates only to the Question-based indicator. This process accepts, as aparameter, the specific question to be observed. In this case, theQuestID would point to the Instrument Question 920 whose description is“Current School Attendance Status (less than 10 is Satisfactory)”. Thisprocess searches through all assessments that contain answers to thisparticular Instrument Question 920.

[0227] To illustrate the navigation within the database corresponding toprocess Step 2_Quest Get Answers 1320, refer to the data model 2000shown in FIG. 20.

[0228]FIG. 16, applies to Instrument-based indicators, so is notrelevant for this indicator type. The third step is illustrated by FIG.17 and shown generally by numeral 1700. This diagram provides anoverview of the process Step 3 Join Assessments to Answers and Scores1340. Process 1340 joins the date-constrained assessment information,found in the interim table tblEvalFirstLastAssts 1211, with thequestion-constrained answer information, found in tblEvalAsstAnswers1212, and stores the information in a third interim table,tblEvalFirstLastAsstsWithAnswersScores 1213.

[0229] The fourth step, shown in FIG. 18 and depicted by numeral 1800,shows how the process Step 4 Join Answered/Scored Assessments toDemographic Profile Info 1350 joins the information intblEvalFirstLastAsstsWithAnswersScores 1213 with demographic informationassociated with the youth who was the subject of the assessments. Thisinformation is stored in the tabletblEvalDemogFirstLastAsstsWithAnswersScores 1214.

[0230] To illustrate the navigation within the database corresponding toprocess Step 4 Join Answered/Scored Assessments to Demographic ProfileInfo, refer to the data model 2400 shown in FIG. 24.

[0231] Each of the three output samples are drawn from the final tabletblEvalDemogFirstLastAsstsWithAnswersScores 1214. See FIGS. 52 to 54,shown generally by numerals 5200 to 5400, respectively, to see examplesof the three types of output. Also refer to FIGS. 44 and 45 to seeexamples of how these three types of output might be invoked. In thisexample, 5200 is invoked by 4440, 5300 by 4450, and 5400 by 4560.

[0232] Measuring the Second Indicator—Assessment Instrument-based—“LOSI”Surrogate Score

[0233] The first step, i.e. Step 1 Get Paired Assessments, is identicalto that described in the Question-based indicator measurement above. Thesecond step is described by FIG. 16, shown generally by numeral 1600. Itdescribes the process Step 2_Inst Get Scores and Interpretation 1330. Itrelates only to the Instrument-based indicator. This process accepts, asa parameter, the specific instrument to be observed. In this case, theInstID would point to the Assessment Instrument 870 whose description is“LOSI Assessment of Risks and Needs.” This process searches through allassessments that contain answers to this particular AssessmentInstrument 870.

[0234] To illustrate the navigation within the database corresponding toprocess Step 2_Inst Get Scores and Interpretation 1330, refer to thedata model 2100 shown in FIG. 21. The third step, i.e. Step 3 JoinAssessments to Answers and Scores 1340, is identical to that describedin the Question-based indicator measurement above.

[0235] To illustrate the navigation within the database corresponding toprocess Step 3 Join Assessments to Answers and Scores 1340, for theInstrument-based indicator measurement, refer to the data model 2200shown in FIG. 22.

[0236] The fourth step, i.e. Step 4 Join Answered/Scored Assessments toDemographic Profile Info 1350, is identical to that described in theQuestion-based indicator measurement above. To illustrate the navigationwithin the database corresponding to process Step 4 Join Answered/ScoredAssessments to Demographic Profile Info 1350, for the Instrument-basedindicator measurement, refer to the data model 2300 shown in FIG. 23.

[0237] Each of the three output samples are drawn from the final tabletblEvalDemogFirstLastAsstsWithAnswersScores 1214. See FIGS. 55 to 57,shown generally by numerals 5500 to 5700, respectively, to see examplesof the three types of output. Also refer to FIG. 46 to see examples ofhow these three types of output might be invoked. In this example, 5500is invoked by 4640, 5600 by 4650, and 5700 by 4690.

[0238] The present invention permits more complicated indicatormeasurements and other types of analysis for the purpose of programevaluation. It can be contemplated that the following factors might alsobe taken into consideration in program evaluation: the individualproviders associated; the program components, interventions and servicesreceived; the amount of funding; the participation termination reason;the number of contacts made; the socioeconomic characteristics of theparticipant; or the living situation of the individual participant.

[0239] Other levels of aggregation are also available. Some examplesare: analyses could be conducted by Program 510, Agency 730, or Fund760; program periods might be more precisely specified; statisticalsamples might specify participants who received specific program,components, services or interventions; and control groups not receivingservices could be compared with groups that did receive services. Thedata model described, collectively, in FIGS. 5 through 10 encompasseseach of the above variations, and could easily accommodate more complexanalysis specifications.

[0240] While the exemplary embodiment of the invention has beendescribed with a degree of particularity, it is the intent that theinvention include all modifications and alterations from the discloseddesign falling within the spirit or scope of the appended claims.

I claim:
 1. A method for performing an aggregate assessment of a groupof subjects given subject-specific assessments performed usingassessment instruments and data-validation rules comprising the stepsof: a) providing, within an assessment-instrument data structure, ameans of representing a set of questions, answer restrictions, andquestion-answer-validation rules; b) providing, within saidassessment-instrument data structure, a means of representing aplurality of assessment instruments, storing, for each said assessmentinstrument, an assessment-instrument key and a plurality of links toassociated questions; c) providing, within a subject data structure, ameans of representing a plurality of subjects, said data structurestoring, for a each said subject, a subject key and a plurality ofsubject attributes; d) providing, within an assessment-event datastructure, a means of representing a plurality of said assessmentevents, said data structure storing, for each said assessment event, anassociated said assessment-event key, associated said subject andassessment-instrument keys, as well as a reference to a point in time;e) storing, within said assessment-event data structure, for each saidassessment event, the assessment results including of a plurality ofvalidated answers to a plurality of associated said questions; and f)deriving, for any group of subjects, assessment results for said groupby aggregating said assessment results from subjects within said groupand utilizing linked data within said data structures.
 2. The method ofclaim 1 wherein a fixed and minimal number of flexible general-purposedata structures are used to represent an unlimited number of assessmentinstruments, and assessment-instrument definition is data-driven anduser-maintained.
 3. The method of claim 1 further including anassessment-instrument-definition user interface that permits users tocreate answers, questions and other assessment-definition elements, suchas question types, answer weights, and attributes such as multi-responseand required, the end result being the creation of assessmentinstruments whose answers can be analyzed in aggregate.
 4. The method ofclaim 1 further including an assessment-event user interface, the methodcomprising: a) using the interface to administer a plurality of saidassessment events; b) using the assessment-instrument definitioninformation residing in the said assessment-instrument data structure toenforce question-answer integrity; c) using, for a given assessmentevent, said assessment-event user interface to display a plurality ofassociated said questions, to capture and validate associated saidsubject's given answer(s), and to store said answers in saidassessment-event data structure.
 5. The method of claim 4 wherein theprocess for answering a given question during an assessment event uses aform whose controls for collecting said subject's answer vary byquestion type, said controls being defined to ensure question-answerintegrity
 6. The method of claim 4 wherein an alternate set of forms canbe used for assessment event administration, in the situation where theassociated said assessment instrument is of simple construct, an examplebeing said instruments which have only single-question question groups.7. The method of claim 1 further including an assessment-event userinterface, the method comprising: a) utilizing the interface toadminister a plurality of said assessment events; and b) utilizing, fora given assessment event, said assessment-event user interface todisplay a plurality of associated said questions, to capture andvalidate associated said subject's given answer(s), and to store saidanswers in said assessment-event data structure.
 8. The method of claim1 further comprising an aggregate-assessment user interface, the methodcomprising the steps of: a) collecting scenario parameters such assubject grouping parameters, analysis-horizon parameters, assessmentinstruments or questions to analyze, output medium, output of graphingformat; and b) utilizing a standard set of queries to operate againstthe said assessment data structures, using the inputted parameters,deriving and presenting aggregate assessment results.
 9. The method ofclaim 1 wherein subject-specific assessment results are provided by anexternal data source, rather than through the standard assessment-eventuser interface or defined within hard-coded assessment instruments 10.The method of claim 10 wherein criminal activity is anexternally-provided assessment result that is used in measuringrecidivism rates for various subject populations
 11. The method of claim1 where special types of assessment instruments have an aggregate scoreassociated with each assessment event that uses that instrument, themethod comprising the steps of: a) defining numeric questions that areto be used in the assessment instrument; b) defining the upper and lowerbounds that the associated answers can take on; c) define the set ofintervals that partition the range of values derived from summing thelower and upper bounds of the said instrument's questions; d)associating, with each said interval, an interpretation; and e) usingsuch aggregate-score instruments, having the system automatically assignan associated score interpretation to the associated events.
 12. Themethod of claim 1 where static subject attributes include sex, race,date of birth, and such attributes are used for subject grouping andanalysis scenarios
 13. The method of claim 1 where dynamic subjectattributes can be captured within assessment events, the methodcomprising the steps of: a) defining dynamic subject attributes asquestions, said attributes containing, for example, marital status, jobinfo, school info, and address; b) creating an assessment instrumentwhich includes said variables; c) utilizing the said assessmentinstrument to conduct assessment events, capturing the dynamicattributes; and d) utilizing assessment instruments, with date stamp, tocapture dynamic subject attributes, such as address, school, job, andmarital status.
 14. A method for tracking program participation withinprograms that provide services to program participants, wherein theparticipation experience includes service events and said assessmentevents, and assessment data has been stored in assessment-instrument andassessment-event data structures, the method comprising the steps of: a)providing a means of representing a plurality of programs having varyingprogram components and services, storing a unique program key and aplurality of program attributes in a program-definition data structure;b) providing a means of representing a plurality ofprogram-participation experiences, storing, for each saidprogram-participation experience, a unique program-participation key, anassociated program key, an associated program-participant key, and areference to a participation time period within a program-participationdata structure; and c) representing, also within saidprogram-participation data structure, for a given saidprogram-participation experience, a plurality of program-participationactivities and events, including said assessment events, wherein eachsaid event is linked to the said experience by saidprogram-participation key.
 15. The method of claim 14, wherein casemanagement of program participants can be facilitated by the capturedprogram-participation and assessment information.
 16. The method ofclaim 15 wherein the method is used for case management purposes, themethod comprising: a) defining activities to be tracked; b) defining theassessment instruments to be used for assessing program participant'sprogress; c) tracking, for each said program participant, the activitiesin which said participant was involved; d) conducting assessments, foreach program participant, assessing participant outcomes; and e)reporting participant-specific information about program participationand assessment results to ascertain relationships between participationand participant outcomes and behavior.
 17. A method for assessing theeffectiveness of programs that provide services to program participantswherein assessment questions have been stored in anassessment-instrument data structure and validated assessment resultsfor program participants have been stored as assessment events in anassessment-event data structure, and are analyzed, in aggregate,relative to program-participation experiences, the method comprising thesteps of: a) providing a means of representing a plurality of programshaving varying program components and services, storing a unique programkey and a plurality of program attributes in a program-definition datastructure; b) providing a means of representing a plurality ofprogram-participation experiences, storing, for each saidprogram-participation experience, a unique program-participation key, anassociated program key, an associated program-participant key, and areference to a participation time period within a program-participationdata structure; c) representing, also within the saidprogram-participation data structure, for a given saidprogram-participation experience, a plurality of program-participationactivities and events, including said assessment events, wherein eachsaid event is linked to the said experience by saidprogram-participation key; and d) for any said program having outcomeindicators represented in said assessment instruments, selecting the setof said assessment events associated with said instrument(s) and withsaid program's said participants, and deriving program-level assessmentresults by aggregating said assessment results of said assessmentevents.
 18. The method of claim 17 wherein programs can utilize saidassessment instruments as one measure of program effectiveness, themethod further comprising the steps of: a. determining outcomeindicators that measure their program effectiveness; b. incorporatingthose outcome indicators into the said assessment instruments bydefining said outcome indicators as questions or as aggregate-scoreassessment instruments; c. administering assessment events to itsparticipants using the said outcome-indicator assessment instruments;and d. aggregating the results from said assessment events to deriveuseful information about the outcomes of the program
 19. The method ofclaim 18 wherein the program-level outcome measures derived fromsubject-specific assessment results can be further aggregated, usingrelationships between program and program provider, program provider andcommunity, community and region, etc., this higher-level aggregationaccomplished by the utilization of standardized instruments, containingstandardized questions whose answers use standardized answers, such asDSM-IV diagnosis codes
 20. The method of claim 17 further comprising anprogram-evaluation user interface, the method comprising the steps of:c) collecting scenario parameters such as program to be evaluated,participant grouping parameters, analysis-horizon parameters, assessmentinstruments or questions to analyze, comparison parameters, outputmedium, output of graphing format; and d) utilizing a standard set ofqueries to operate against the said assessment data structures and theprogram-participation data structures, using the inputted parameters,deriving and presenting aggregate assessment results.
 21. A computerreadable medium having computer executable instructions stored thereonfor performing method steps for performing an aggregate assessment of agroup of subjects given subject-specific assessments performed usingassessment instruments and data-validation rules, the method stepscomprising: a) providing, within an assessment-instrument datastructure, a means of representing a set of questions, answerrestrictions, and question-answer-validation rules; b) providing, withinsaid assessment-instrument data structure, a means of representing aplurality of assessment instruments, storing, for each said assessmentinstrument, an assessment-instrument key and a plurality of links toassociated questions; c) providing, within a subject data structure, ameans of representing a plurality of subjects, said data structurestoring, for a each said subject, a subject key and a plurality ofsubject attributes; d) providing, within an assessment-event datastructure, a means of representing a plurality of said assessmentevents, said data structure storing, for each said assessment event, anassociated said assessment-event key, associated said subject andassessment-instrument keys, as well as a reference to a point in time;e) storing, within said assessment-event data structure, for each saidassessment event, the assessment results including of a plurality ofvalidated answers to a plurality of associated said questions; and f)deriving, for any group of subjects, assessment results for said groupby aggregating said assessment results from subjects within said groupand utilizing linked data within said data structures.
 22. A computerreadable medium having computer executable instructions stored thereonfor performing method steps for assessing the effectiveness of programsthat provide services to program participants wherein assessmentquestions have been stored in an assessment-instrument data structureand validated assessment results for program participants have beenstored as assessment events in an assessment-event data structure, andare analyzed, in aggregate, relative to program-participationexperiences, the method steps comprising: a) providing a means ofrepresenting a plurality of programs having varying program componentsand services, storing a unique program key and a plurality of programattributes in a program-definition data structure; b) providing a meansof representing a plurality of program-participation experiences,storing, for each said program-participation experience, a uniqueprogram-participation key, an associated program key, an associatedprogram-participant key, and a reference to a participation time periodwithin a program-participation data structure; c) representing, alsowithin the said program-participation data structure, for a given saidprogram-participation experience, a plurality of program-participationactivities and events, including said assessment events, wherein eachsaid event is linked to the said experience by saidprogram-participation key; and d) for any said program having outcomeindicators represented in said assessment instruments, selecting the setof said assessment events associated with said instrument(s) and withsaid program's said participants, and deriving program-level assessmentresults by aggregating said assessment results of said assessmentevents.
 23. A computer readable medium having computer-executableinstructions stored thereon for assessing the effectiveness of programsthat provide services to program participants wherein assessmentquestions have been stored in an assessment-instrument data structureand validated assessment results for program participants have beenstored as assessment events in an assessment-event data structure, andare analyzed, in aggregate, relative to program-participationexperiences, the method comprising the steps of: a) providing a means ofrepresenting a plurality of programs having varying program componentsand services, storing a unique program key and a plurality of programattributes in a program-definition data structure; b) providing a meansof representing a plurality of program-participation experiences,storing, for each said program-participation experience, a uniqueprogram-participation key, an associated program key, an associatedprogram-participant key, and a reference to a participation time periodwithin a program-participation data structure; c) representing, alsowithin the said program-participation data structure, for a given saidprogram-participation experience, a plurality of program-participationactivities and events, including said assessment events, wherein eachsaid event is linked to the said experience by saidprogram-participation key; and d) for any said program having outcomeindicators represented in said assessment instruments, selecting the setof said assessment events associated with said instrument(s) and withsaid program's said participants, and deriving program-level assessmentresults by aggregating said assessment results of said assessmentevents.